Alert Actioning and Machine Learning Feedback

ABSTRACT

Some aspects of the present disclosure relate to systems, methods, and computer-readable media for configuring a conduct surveillance system. In one example implementation, a computer implemented method includes: receiving at least one alert from a conduct surveillance system, where the at least one alert represents a potential violation of a predetermined policy, where the predetermined policy includes a scenario, a target population, and a workflow; determining whether each of the at least one alert represents an actual violation of the predetermined policy; calculating a metric based on the actual violations and the potential violations where the metric includes a number of false positives associated with the at least one alert or the number of false negatives associated with the at least one alert; and changing at least one of the scenario, the target population, or the workflow based on the calculated metric.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and benefit of U.S. provisionalpatent application Ser. No. 63/160,780 filed Mar. 13, 2021 and U.S.provisional patent application Ser. No. 63/162,829 filed Mar. 18, 2021,which are hereby fully incorporated by reference and made a part hereof.

BACKGROUND

The present disclosure generally relates to monitoring communicationsfor activity that violates ethical, legal, or other standards ofbehavior and poses risk or harm to institutions or individuals. The needfor detecting violations in the behavior of representatives of aninstitution has become increasingly important in the context ofproactive compliance, for instance. In the modern world of financialservices, there are many dangers to large institutions from a complianceperspective, and the penalties for non-compliance can be substantial,both from a monetary standpoint and in terms of reputation. Financialinstitutions are coming under increasing pressure to quickly identifyunauthorized trading, market manipulation and unethical conduct withintheir organization, for example, but often lack the tools to do soeffectively.

Moreover, systems and methods for monitoring communications can be tunedor adapted to detect types of violations of behavior, or to increase theaccuracy of those systems and methods. Advanced systems and methods formonitoring communication can be based on complicated models, includingmachine learning models and other techniques. Therefore it can bedifficult for a user of a system or method to tune or adapt that systemor method.

Thus, among other needs, there exists a need for effectiveidentification of activity that violates ethical, legal, or otherstandards of behavior and poses risk or harm to institutions orindividuals from electronic communications. Furthermore, there exists aneed for effective ways to improve the identification of violationconditions and effective ways to configure systems to identify violationconditions. It is with respect to these and other considerations thatthe various embodiments described below are presented.

SUMMARY

Embodiments of the present disclosure are directed generally towardsmethods, systems, and computer-readable storage medium relating to, insome embodiments, an intuitive review and investigation tool designed tofacilitate efficient and defensible reviews of electronic communications(e.g., messages) by analysts (sometimes referred to herein as “users”).In certain implementations, users are empowered to quickly target riskareas and increase accuracy of reviews as a result of a flexibleworkflow for evaluating alerts. Users can also have a more granularfeedback loop for alerts as an input into reporting and model training(see FIG. 13, for example). Actioning can be performed on communicationsat the hit, alert, and/or message level to illustrate progress ofreviewed communications and streamline business processes (see FIG. 13,for example).

A streamlined actioning workflow can allow users to easily close alertsand add relevant context (for example, person of interest and comments)to elevated alerts requiring further review (see FIG. 25, for example).Alerts assigned to a user can be accessed from a user dashboard, wherethe user can also see the total messages awaiting review (see FIG. 26,for example).

In one aspect, the present disclosure relates to a computer-implementedmethod, which, in one embodiment, receiving at least one alert from aconduct surveillance system, where the at least one alert represents apotential violation of a predetermined standard and where the conductsurveillance system generates the alerts in response to an electroniccommunication between persons matching a violation of a predeterminedpolicy, where the predetermined policy includes a scenario, a targetpopulation, and a workflow; determining whether each of the at least onealert represents an actual violation of the predetermined policy;calculating a metric based on the actual violations and the potentialviolations where the metric includes a number of false positivesassociated with the at least one alert or the number of false negativesassociated with the at least one alert; and changing at least one of thescenario, the target population, or the workflow based on the calculatedmetric.

In some embodiments of the present disclosure, the scenario includes amachine learning classifier, and where determining whether the at leastone alert represents an actual violation includes labeling the at leastone alert and using the labeled at least one alert to train the machinelearning classifier.

In some embodiments of the present disclosure, the metric is displayedto a user.

In some embodiments of the present disclosure, the scenario includes alexicon, and where the lexicon represents one or more terms or regularexpressions.

In some embodiments of the present disclosure, changing the scenarioincludes changing the lexicon by adding or removing terms or regularexpressions from the lexicon.

In some embodiments of the present disclosure, the computer implementedmethod includes, in response to determining that the at least one alertrepresents an actual violation, actioning the alert.

In some embodiments of the present disclosure, actioning the alertincludes receiving a user input from the user interface representingwhether the at least one alert represents an actual violation.

In some embodiments of the present disclosure, the target populationincludes a domain exclusion list and where changing the targetpopulation includes changing the domain exclusion list.

In some embodiments of the present disclosure, the electroniccommunication includes metadata, the scenario includes rules forfiltering the electronic communication based on the metadata, and wherechanging the scenario includes changing the rules for filtering theelectronic communications based on the metadata.

In another aspect, the present disclosure relates to a system, which inone embodiment includes: at least one processor; at least one memorystoring computer readable instructions configured to cause the at leastone processor to perform functions for creating and/or evaluatingmodels, scenarios, lexicons, and/or policies, where the functionsinclude: receiving data associated with at least one of text data, modeltraining, lexicons, scenarios, and policies, where the functions forcreating and/or evaluating models comprise creating at least onescenario based on at least one of the models, lexicons, and non-languagefeatures; creating one or more policies mapping to the at least onescenario and a population; upon receiving an alert that a policy matchoccurs, triggering an alert indicating, to a user, that a policy matchhas occurred which requires a user action, where a policy corresponds toactions that violate at least one of a combination of signals andmetrics, a population, and workflow.

In some embodiments of the present disclosure, the model trainingincludes training at least one model configured to analyze the text datafrom one or more electronic communications between at least two persons.

In some embodiments of the present disclosure, the user action includesreview and interaction by a user via a user interface.

In some embodiments of the present disclosure, the model trainingincludes evaluating the model against established datasets.

In some embodiments of the present disclosure, the alert to the user isevaluated by the user and a corresponding user decision is made toconfirm or deny accuracy of the alert.

In some embodiments of the present disclosure, the user decision isprovided into a feedback loop, and where the feedback loop is configuredto improve the model training.

In some embodiments of the present disclosure, the user decision isprovided into the feedback loop and where the feedback loop isconfigured to improve the lexicons, scenarios, or policies.

In some embodiments of the present disclosure, the feedback loop isconfigured to change a lexicon.

In some embodiments of the present disclosure, changing the lexiconincludes configuring the lexicon so that it includes or excludes termsor regular expressions.

In some embodiments of the present disclosure, the feedback loop isconfigured to measure the rate of false positives and to change one ormore of the lexicons, scenarios, and policies based on the rate of falsepositives.

In some embodiments of the present disclosure, the scenario includesBoolean operators, and where the feedback loop is configured to changeone or more of the Boolean operators.

In some embodiments of the present disclosure, the feedback loop isconfigured to monitor the rate of false positives over a period of time,and change one or more of the lexicons, scenarios, and policies based onthe rate of false positives over the period of time.

In yet another aspect, the present disclosure relates to anon-transitory computer-readable medium storing instructions which, whenexecuted by one or more processors, cause a computing device to performspecific functions. The functions performed include receiving at leastone alert from a conduct surveillance system, where the at least onealert represents a potential violation of a predetermined standard andwhere the conduct surveillance system generates the alerts in responseto an electronic communication between persons matching a violation of apredetermined policy, where the predetermined policy includes ascenario, a target population, and a workflow; determining whether eachof the at least one alert represents an actual violation of thepredetermined policy; calculating a metric based on the actualviolations and the potential violations where the metric includes anumber of false positives associated with the at least one alert or thenumber of false negatives associated with the at least one alert; andchanging at least one of the scenario, the target population, or theworkflow based on the calculated metric.

The following provides a non-limiting discussion of some exampleimplementations of various aspects of the present disclosure. Someaspects and embodiments disclosed herein may be utilized for providingadvantages and benefits in the area of communication surveillance forregulatory compliance. Some implementations can process allcommunications, including electronic forms of communications such asinstant messaging (or “chat”), email, voice, and/or social networkmessaging to connect and monitor an organization's employeecommunications for regulatory and corporate compliance purposes. Someembodiments of the present disclosure unify detection, user interfaces,behavioral models, and policies across all communication data sources,and can provide tools for compliance analysts in furtherance of thesefunctions and objectives. Some implementations can proactively analyzeusers' actions to identify breaches such as unauthorized activities thatare against applicable policies, laws, or are unethical, through the useof natural language processing (NLP) models. The use of these models canenable understanding content of communications and map signals tobehavioral profiles in order to locate high-risk individuals.

Other aspects and features according to the example embodiments of thepresent disclosure will become apparent to those of ordinary skill inthe art, upon reviewing the following detailed description inconjunction with the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale.

FIGS. 1A-1C illustrate methods according to various aspects of thepresent disclosure. FIG. 1A illustrates a method for creating alertsbased on a policy match according to one embodiment of the presentdisclosure. FIG. 1B illustrates a method of configuring a computersystem to detect violations in a target dataset according to oneembodiment of the present disclosure. FIG. 1C illustrates a method forincreasing the accuracy of a conduct surveillance system according toone embodiment of the present disclosure.

FIGS. 2A-2C illustrate various aspects of the present disclosure. FIG.2A illustrates various aspects of displayed events, properties, andcommunications data, in accordance with one or more embodiments of thepresent disclosure. FIGS. 2B and 2C illustrate various aspects ofpolicies, including scenario, population, and workflow, in accordancewith one or more embodiments of the present disclosure.

FIG. 3 is a diagram relating to workflows in accordance with one or moreembodiments of the present disclosure.

FIG. 4 illustrates various aspects displayed to a user interface,including elements of a graphical user interface, with a sidebar,content, and aside areas, in accordance with one or more embodiments ofthe present disclosure.

FIG. 5 illustrates a visual view including various data representationsbeyond simple text, in accordance with one or more embodiments of thepresent disclosure.

FIG. 6 illustrates aspects of knowledge tasks, in accordance with one ormore embodiments of the present disclosure.

FIG. 7 illustrates a profile view corresponding to a particular entity,in accordance with one or more embodiments of the present disclosure.

FIG. 8 illustrates a profile view and particularly labels an “aside”section of a displayed graphical user interface, in accordance with oneor more embodiments of the present disclosure.

FIG. 9 illustrates various aspects of alerts, hits, and actions, inaccordance with one or more embodiments of the present disclosure.

FIG. 10 illustrates various aspects of alert hit previews and listcards, in accordance with one or more embodiments of the presentdisclosure.

FIG. 11 illustrates various aspects of metrics and tabs, in accordancewith one or more embodiments of the present disclosure.

FIG. 12 is a computer architecture diagram showing a general computingsystem capable of implementing one or more embodiments of the presentdisclosure described herein.

FIG. 13 is a flow diagram illustrating components and operations of afeedback loop and system in accordance with one embodiment of thepresent disclosure.

FIG. 14 illustrates various aspects of a model dashboard, in accordancewith one or more embodiments of the present disclosure.

FIG. 15 illustrates various aspects of a model dashboard withstatistics, in accordance with one or more embodiments of the presentdisclosure.

FIG. 16 illustrates various aspects of lexicon evaluation, in accordancewith one or more embodiments of the present disclosure.

FIG. 17 illustrates various further aspects of lexicon evaluationincluding a confusion matrix, in accordance with one or more embodimentsof the present disclosure.

FIG. 18 illustrates various further aspects of lexicon evaluation, inaccordance with one or more embodiments of the present disclosure.

FIG. 19 illustrates various further aspects of lexicon evaluation, inaccordance with one or more embodiments of the present disclosure.

FIG. 20 illustrates various aspects of scenarios in accordance with oneor more embodiments of the present disclosure.

FIG. 21 illustrates various aspects of policy administrationfunctionality in accordance with one or more embodiments of the presentdisclosure.

FIG. 22 illustrates a user interface for accessing a repository inaccordance with one or more embodiments of the present disclosure.

FIGS. 23A-23F illustrates user interfaces for configuring a scenario inaccordance with one or more embodiments of the present disclosure. FIG.23A illustrates a user interface for viewing one or more datasets. FIG.23B illustrates a user interface for labeling a dataset. FIG. 23Cillustrates an annotation applied to a dataset and an interface forapplying labels to a dataset. FIG. 23D illustrates a user interface forconfiguring a lexicon to be applied to the dataset. FIG. 23E illustratesa user interface for evaluating a lexicon. FIG. 23F illustrates ascenario created using the lexicon that was configured in the interfaceshown in FIG. 23E.

FIG. 24 illustrates various aspects of actioning communications inaccordance with one or more embodiments of the present disclosure.

FIG. 25 illustrates various aspects of actioning communications inaccordance with one or more embodiments of the present disclosure.

FIG. 26 illustrates various aspects of actioning communications inaccordance with one or more embodiments of the present disclosure.

DETAILED DESCRIPTION

Although example embodiments of the present disclosure are explained indetail herein, it is to be understood that other embodiments arecontemplated. Accordingly, it is not intended that the presentdisclosure be limited in its scope to the details of construction andarrangement of components set forth in the following description orillustrated in the drawings. The present disclosure is capable of otherembodiments and of being practiced or carried out in various ways.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless the context clearly dictates otherwise.

By “comprising” or “containing” or “including” is meant that at leastthe named compound, element, particle, or method step is present in thecomposition or article or method, but does not exclude the presence ofother compounds, materials, particles, method steps, even if the othersuch compounds, material, particles, method steps have the same functionas what is named.

In describing example embodiments, terminology will be resorted to forthe sake of clarity. It is intended that each term contemplates itsbroadest meaning as understood by those skilled in the art and includesall technical equivalents that operate in a similar manner to accomplisha similar purpose. It is also to be understood that the mention of oneor more steps of a method does not preclude the presence of additionalmethod steps or intervening method steps between those steps expresslyidentified. Steps of a method may be performed in a different order thanthose described herein without departing from the scope of the presentdisclosure. Similarly, it is also to be understood that the mention ofone or more components in a device or system does not preclude thepresence of additional components or intervening components betweenthose components expressly identified.

Definitions

The following discussion provides some descriptions and non-limitingdefinitions, and related contexts, for terminology and concepts used inrelation to various aspects and embodiments of the present disclosure.

An “event” can be considered any object with a fixed time, and an eventcan be observable data that happens at a point in time, for example anemail, a badge swipe, a trade (e.g., trade of a financial asset), or aphone call (see also the illustration of FIG. 2A).

A “property” relates to an item within an event that can be uniquelyidentified, for example metadata (see also illustration of FIG. 2A).

A “communication” or “electronic communication” can be any event withlanguage content, for example email, chat, a document, social media, ora phone call (see also illustration of FIG. 2A). An electroniccommunication may also include, for example, audio, SMS, and/or video. Acommunication may additionally or alternatively be referred to hereinas, or with respect to, a “comm” (or “comms”), message, container,report, or data payload.

A “metric” can be a weighted combination of factors to identify patternsand trends (e.g., a number-based value to represent behavior or intentfrom a communication). Examples of metrics include sentiment, flightrisk, risk indicator, and responsiveness score. A metric mayadditionally or alternatively be referred to herein as, or with respectto, a score, measurement, or rank.

A “post” can be an identifier's contribution within a communication, forexample a single email within a thread, a single chat post, a continuousburst of communication from an individual, or a single social media post(see also illustration of FIG. 2A). A post can be considered as anindividual's contribution to a communication.

A “conversation” can be a group of semantically related posts, forexample the entirety of an email with replies, a thread, or alternativea started and stopped topic, a time-bound topic, and/or a post with theother post (replies). Several posts can make up a conversation within acommunication.

A “signal” can be an observation tied to a specific event that isidentifiable, for example rumor language, wall crossing, or language ofinterest.

A “policy” can be a scenario applied to a population with a definedworkflow. A policy may be, for instance, how a business chooses tohandle specific situations, for example as it may relate to ongoing dealmonitoring, disclaimer adherence, and/or anti money laundering (AML)monitoring. As used herein, a policy may additionally or alternativelybe referred to as, or with respect to, a “KI” or “key indicator”, orrules engine. As illustrated in FIGS. 2B and 2C, in some embodiments apolicy can be comprised of three items: a scenario as a combination ofsignals and metrics (as an example of usage, using NLP signals andmetrics to discover intellectual property (IP) theft language orbehaviors); a population, as the target population over which to lookfor the scenario (e.g., sales team(s), department(s), or group(s) ofpersons); and workflow, as actions taken when a scenario triggers over apopulation (e.g., alert generation).

An “alert” can indicate to a user that a policy match has occurred whichrequires action (sometimes referred to herein with respect to“actioning” an alert), for example a scenario match. A signal thatrequires review can be considered an alert. As an example, an indicationof intellectual property theft may be found in a chat post with languagethat matches the scenario, on a population that needs to be reviewed.

A “manual alert” can be an alert added to a communication from a user,not generated from the system. A manual alert may be used, for example,when a user needs to add an alert to language or other factors forfurther review.

A “hit” can be an exact signal that applies to a policy on events, forexample an occurrence of the language “I'm taking clients with me when Ileave”, a behavior pattern change, and/or a metric change. As usedherein, a hit may additionally or alternatively be referred to hereinas, or with respect to, a “KI” (“key indicator”), event, and/orhighlight.

A “review” can be the act of a user assigning actions on hits, alerts,or communications.

A “tag” can be a label attached to a communication for the purpose ofidentification or to give other information, for example a new featureset that will enable many workflow practices.

A “knowledge graph” can be a representation of all of the signals,entities, topics, and relationships in a data set in storage. Knowledgegraphs can communications, some of which may contain alerts for a givenpolicy. Other related terms may include a “knowledge base.” In someembodiments, a knowledge graph can be a unified knowledgerepresentation.

A “personal identifier” can be any structured field that can be used todefine a reference or entity, for example “jeb@jebbush.com”, “@CMcK”,“EnronUser1234”, or “(555) 336-2700” (i.e., a personal identifier caninclude email, a chat handle, or a phone number). As used herein, a hitmay additionally or alternatively be referred to herein as, or withrespect to, an “entity ID”.

A “mention” can be any descriptive string that is able to be referencedand/or extracted, for example “He/Him”, “The Big Blue”, “Enron”, or“John Smith”. Other related terms may include “local coreference.”

An “entity” can be an individual, object, and/or property IRL, and canhave multiple identifiers or references, for example John Smith, IBM, orEnron. Other related terms may include profile, participant, actor,and/or resolved entity.

A “relationship” can be a connection between two or more identifiers orentities, for example “works in” department, person-to-person,person-to-department, and/or company-to-company. Other related terms mayinclude connections via a network graph.

The following discussion includes some descriptions and non-limitingdefinitions, and related contexts, for terminology and concepts that mayparticularly relate to workflows in accordance to one or moreembodiments of the present disclosure, some of which may be furtherunderstood by reviewing the diagram of FIG. 3.

A “smart queue” can be a saved set of search modifiers with an owner anddefined time, for example, a daily bribery queue, an action pendingqueue, an escalation queue, or any shared/synced list. As used herein, asmart queue may additionally or alternatively be referred to herein as,or with respect to an action pending queue, analyst queue, or scheduledsearch.

A “saved search” can be a saved set of search modifiers with no owner,for example a monthly QA check, an investigation search, or anirregularly used search. As used herein, a saved search may additionallyor alternatively be referred to herein as, or with respect to a searchcopy or a bookmark.

The following discussion includes some descriptions and non-limitingdefinitions, and related contexts, for terminology and concepts that canrelate to a graphical user interface (and associated example views asoutput to a user) that can be used by a user to interact with,visualize, and perform various functionalities in accordance to one ormore embodiments of the present disclosure.

A “sidebar” can be a global placeholder for navigation and branding(see, e.g., illustrations in FIG. 4.

“Content” as shown and labeled in, for example, FIG. 4, identifies whereprimary content will be displayed.

An “aside” as shown and labeled in, for example, FIG. 4, is a locationfor supportive components that affect the content or provide additionalcontext. Further related aspects of “aside” are shown in the example ofFIG. 8. An aside can be a column of components that support, define, ormanipulate the content area.

A “visual view” as illustrated in, for example, FIG. 5, can include achart, graph, or data representation that is beyond simple text, forexample communications (“comms”) over time, alters daily, queueprogress, and/or relationship metric(s). As used herein, visual viewsmay additionally or alternatively be referred to herein as, or withrespect to charts or graphs.

A “profile” can be a set of visuals filtered by an identifier or entity,for example by a specific person's name, behavior analytics, anorganization's name, or QA department. As used herein, profiles mayadditionally or alternatively be referred to herein as, or with respectto relationship(s) or behavior analytics.

Now also referring to the diagram of FIG. 6, smart queues can enableteams to work to accelerate “knowledge tasks”. Signals that requirereview (i.e., alerts), comprise monitoring. These can be from externalsystems. Knowledge tasks can provide feedback via a “learning loop” intomodels.

Now also referring to the view in the illustration of FIG. 7, aparticular profile view can provide insights such as behavioral insightsto, for instance, an entity (here, a particular person). The profile caninclude a unified timeline with hits, and communications. Also, profilescan provide aggregates of/into entities, metrics, visuals, events, andrelationships. As mentioned briefly above and as illustrated in FIG. 8,an aside can be a column of components that support, define, ormanipulate the content area.

Now referring to the view in the illustrations of FIGS. 9 and 10, and asdiscussed in some detail above, an “alert” can be the manifestation of apolicy on events, and a “hit” (or “alert hit”) can be the exact signalthat applies to a policy on events. An “action” can be the label that isapplied to: a single hit; all hits under an alert; or all hits on amessage. A “list card” can be an object that contains a summary of thecontent of a comm in the “list view”, which can be a list of events withcommunications that may have an alert.

Now referring to the view in the illustration of FIG. 11, as discussedin some detail above, a “metric” can be a weighted combination offactors to identify patterns and trends. A “tab” can be an additionalview that can display content related to a current view, for examplesibling content.

The following discussion includes some descriptions and non-limitingdefinitions, and related contexts, for terminology and concepts that mayparticularly relate to machine learning models and the training ofmachine learning models, in accordance with one or more embodiments ofthe present disclosure.

A “hit” can be an exact signal that applies to a policy on events, forexample an occurrence of the language “I'm taking clients with me when Ileave”, a behavior pattern change, and/or a metric change. As usedherein, a hit may additionally or alternatively be referred to hereinas, or with respect to, a “KI” (“key indicator”), event, and/orhighlight.

A “pre-trained model” can be a model that performs a task but requirestuning (e.g., supervision and/or other interaction by an analyst ordeveloper) before production. An “out of the box model” can be a modelthat benefits from, but does not require, tuning before use inproduction. Pre-trained models and out of the box models can be part ofthe building blocks for a policy. As used herein, a pre-trained modelmay additionally or alternatively be referred to herein as, or withrespect to, “KI engines” or “models”.

In some embodiments, the present disclosure can provide for implementinganalytics using “supervised” machine learning techniques (herein alsoreferred to as “supervised learning”). Supervised mathematical modelscan encode a variety of different data aspects which can be used toreconstruct a model at run-time. The aspects utilized by these modelsmay be determined by analysts and/or developers, for example, and may befixed at model training time. Models can be retrained at any time, butretraining may be done more infrequently once models reach certainlevels of accuracy.

DESCRIPTION OF EXAMPLE EMBODIMENTS OF PRESENT DISCLOSURE

A detailed description of various aspects of the present disclosure, inaccordance with various example embodiments, will now be provided withreference to the accompanying drawings. The drawings form a part hereofand show, by way of illustration, specific embodiments and examples.

The following provides a non-limiting discussion of some exampleimplementations of various aspects of the present disclosure

In some embodiments, the present disclosure is directed to a system forindicating to a user when a policy match has occurred which requiresaction by the user. The system can include a processor and a memoryconfigured to cause the processor to perform functions for creatingand/or evaluating models, scenarios, lexicons, and/or policies. As anon-limiting example, the processor and memory can be part of thegeneral computing system illustrated in FIG. 12.

Embodiments of the present disclosure can implement the methodillustrated in FIG. 1A The instructions stored on the memory can includeinstructions to receive 102 data associated with text data, modeltraining, lexicons, scenarios and/or policies. Creating and/orevaluating models can include creating a scenario based on the models,lexicons, and non-language features. It should be understood that thescenario can be based on any combination of models, lexicons, andnon-language features. As a non-limiting example, the scenario can bebased on a single model, but multiple lexicons and multiple non-languagefeatures.

As described herein, the model can correspond to a machine learningmodel. In some embodiments, the machine learning model is a machinelearning classifier that is configured to classify text. Additionally,in some embodiments, the model training can include training models foranalysis of text data from one or more electronic communications betweenat least two persons.

The present disclosure contemplates the machine learning trainingtechniques known in the art can be applied to the data disclosed in thepresent disclosure for model training. For example, in some embodiments,the model training can include evaluating the model against establisheddatasets. As another example, the model training can be based on a userinput, for example a user input that labels the data.

The system can be configured to create 104 one or more policies mappingto the scenario and a population. In embodiments with more than onescenario and/or more than one policy, it should be understood that anynumber of scenarios and/or policies can be mapped to one another. Asnon-limiting examples, the system can be configured to map multiplescenarios to multiple policies, or multiple scenarios to the same policyor policies.

When the system receives an alert that a policy match occurs, the systemcan trigger 106 an alert indicating, to a user, that a policy match hasoccurred which requires action. The policy can correspond to actionsthat violate at least one of a combination of signals and metrics, apopulation, and workflow (referred to herein as a “violation”)

Additionally, the present disclosure contemplates that the alerts can bereviewed by the user or by a machine learning model. This review caninclude determining whether the alerts correspond to an actualviolation, and can be used to change the scenario, or change any of theparts of the scenario (e.g. models, lexicons, and non-languagefeatures).

In some embodiments of the present disclosure, a user can review thedata and perform an interaction using a user interface (e.g., agraphical user interface that is part of or operably connected to thecomputer system illustrated in FIG. 12). The action can include reviewand interaction by a user via a user interface, which is optionally partof the computing device in FIG. 12. As a non-limiting example, in someembodiments, the system can provide the alert to the user through theuser interface, and then the user can confirm or deny the accuracy ofthe alert using the user interface. Based on the user input, the systemcan determine whether the alert was a true positive, true negative,false positive, or false negative. The system can use the informationabout the alerts, including whether the alert was a true positive, truenegative, false positive, or false negative, as an input into the systemto improve the operation of the system. This can be referred to as“feedback.” The present disclosure contemplates that the feedback can bean input into the machine learning model to improve the model training(e.g. the information about the alerts is “fed back” into the model totrain the model further). Alternatively or additionally, the presentdisclosure contemplates that the feedback can be used to change otherparameters within the scenario. For example, the feedback can be used toadjust the lexicon or non-language features of the scenario. This caninclude adding or removing terms from the lexicon, or adding/removingnon-language features from the scenario.

As a non-limiting example, a scenario has a pre-trained machine learningmodel, a target lexicon of regular expressions and text, and a targetset of non-language features that includes metadata. In this example,the scenario can be configured to identify communications thatcorrespond to the machine learning model and lexicon, where the metadatashows that the communication is from a time span of the previous twoyears. The system can then produce alerts by determining whether each ofthe communications in the dataset is a policy match with the scenario.The user can review the communications that are a policy match with thescenario, and determine whether each communication is a violation, andinput those results into the system. Then, based on those results, thesystem can be configured to change the scenario to improve theeffectiveness of the scenario. This can include maximizing or improvingcertain measures of accuracy such as the ROC curve described herein, thetrue positive rate, precision, recall, or confusion matrix. As anon-limiting example, this can include changing the scenario to targetmetadata in a shorter timeframe, e.g., by changing it from two years toone year. The system and/or the user can then use one or more of themeasures of accuracy (e.g., the true positive rate) to see if themeasure of accuracy has improved after changing the scenario. Bymonitoring the accuracy of the scenario as the scenario is changed, itis possible to tune the scenario to improve the measures of accuracy.Again, these are merely non-limiting examples of techniques formeasuring the error rate, and it will be understood to one of skill inthe art that any techniques for measuring error rate that are known inthe art can be used in combination with the system and methods disclosedherein.

Embodiments of the present disclosure can also include computerimplemented methods for configuring a computer system to detectviolations in a target dataset.

With reference to FIG. 1B, the method 120 can include receiving 122 dataassociated with an electronic communication. The received data caninclude text data, and optionally metadata that are associated with oneor more communications. As a non-limiting example, the data can includea set of emails, text messages, transcribed phone conversations, orcombinations thereof. This data can also include “metadata” that cancorrespond to any information about the communication that is not foundin the text itself.

At step 124, the received data can be labeled. As described throughoutthe present disclosure, labeling can include applying a label indicatingwhether the one or more communications that are part of the datacorrespond to a violation. Labeling can also include determining whetherthe received data includes a segment of target language, and applying alabel to the parts of the data that contain that segment of targetlanguage. As a non-limiting example, this can include labeling certaincommunications in the dataset that contain the target language.

At step, 128, a machine learning model can be created based on the data.As described elsewhere in the present disclosure, this machine learningmodel can be a machine learning classifier that is configured toclassify text. As a non-limiting example, the present disclosurecontemplates that the model training can include evaluating the modelagainst established datasets. As another non-limiting example, the modeltraining can include training at least one model configured to analyzetext data from one or more electronic communications between at leasttwo persons. Additionally, it should be understood that the machinelearning model can be any of the other machine learning models describedherein, or known in the art.

At step 126, a lexicon can be created for the scenario. As describedthroughout the present disclosure, the lexicon can represent one or moreterms or regular expressions. Optionally, at step 126, the lexicon canbe imported partially or completely from a database, or chosen from alist of pre-generated lexicons by a user.

At step 130, a scenario can be created using the machine learning modelsand the lexicon, where the scenario can represent a violation condition(e.g. a violation of an ethical policy, regulatory policy, rule, lawetc., as described in the other examples herein). The user can createthe scenario by specifying the model or models that are used, as well asthe lexicon or lexicons that are used.

In some embodiments, the scenario can be created 130 using componentsother than just a machine learning model and lexicon. For example, thescenario can include a filter, where the filter can be configured toexclude or include at least part of the dataset based on the data in thedataset. This can include filtering based on data such as metadata.Again, it should be understood that metadata can refer to any of theproperties of a communication that are stored in the data, non-limitingexamples of which are the time sent, time received, type ofcommunication, etc.

The user or system can also specify how the models and lexicons arejoined together. Again, as a non-limiting example, the scenario cancombine one or more models and lexicons using Boolean logic (e.g. AND,OR, NOT, NOR). It should be understood that other logical systems andother logical operators can be used in combination with the methoddisclosed herein.

Optionally, in some embodiments, the scenario can be created based onfeedback from actions the user has taken in response to pervious alerts(described herein as “actioning” the alerts). This can include providinga user decision or user action into a feedback loop that is configuredto improve the model training. As a non-limiting example, this userdecision can include confirming or denying the accuracy of the alert. Insome embodiments, the feedback loop can be configured to improve thelexicons, scenarios, or policies. As yet another non-limiting example,the feedback loop can be configured to change the lexicon, and changingthe lexicon can include changing the lexicon so that it includes orexcludes terms or regular expressions. As another non-limiting example,the scenario can include one or more Boolean operators, and the feedbackloop can be configured to change one or more of those Boolean operators.Furthermore, in some embodiments of the present disclosure, the feedbackloop can be configured to measure the rate of false positives betweenthe actual and potential violations identified by the system, and changeone or more of the lexicons, scenarios, and policies based on the rateof false positives. The feedback loop can also be configured to measurethe rate of false positives over a period of time, and change one ormore of the lexicons, scenarios, and policies based on the rate of falsepositives over the period of time.

It should be understood that the rate of false positives is intendedonly as a non-limiting example, and that the feedback loop can beconfigured to change the scenario, lexicons, and policies based on othermeasurements of error, accuracy, etc. As a non-limiting example,example, based on the actioning, the system can be configured to add orremove lexicons or models from the scenario.

At step 132, the computer system (e.g. the computer system of FIG. 12)can be configured to detect violation conditions in a target datasetusing the scenario. This can include storing the scenario in a computerreadable medium, receiving additional data for review, and determiningwhether the additional data contains communications that match thescenario (i.e. that are a “policy match”).

In some implementations, the scenario can be configured to allow for auser to easily configure the scenario. The system can be configured toprevent a user from changing the machine learning model, but enable theuser to change parameters other than the model. This can allow the userto change the scenario and the type of communications identified by thescenario, without requiring knowledge of the machine learning model, orrequiring that the model undergo retraining before use. In someembodiments of the present disclosure, techniques that can be used toreduce the error rates or increase the accuracy other than changing themodel itself can be referred to as the “augmentation layer.”Non-limiting examples of techniques that can be included in theaugmentation layer include lexicons, domain exclusion lists, andrules-based filter using metadata (e.g., filtering out alerts based onnumber of participants or message directionality). The presentdisclosure contemplates that any or all of the techniques in theaugmentation layer can be adjusted based on the dataset.

Furthermore, the present disclosure contemplates that the scenario canbe stored in a computer readable medium, for example the memoryillustrated in FIG. 12. Similarly, in some embodiments of the presentdisclosure, more than one scenario can be stored in one or more computerreadable medium. The one or more scenarios can be compared to oneanother, and the system can create an output based on the comparison. Asa non-limiting example, the output based on the comparison can show whatparts of the scenario are different, or what parts of the scenario havestayed the same, between the two scenarios. As a non-limiting example,this could include displaying that two scenarios include the samelexicon, but include different models, or different Boolean operators.The output including the difference between the first and secondscenario can also include information about the versions of the twoscenarios.

Additionally, some embodiments of the present disclosure are directed toa computer-implemented method 140 for increasing the accuracy of aconduct surveillance system. With reference to FIG. 1C, the method caninclude receiving 142 at least one alert from a conduct surveillancesystem. As used in the present disclosure, a “conduct surveillancesystem” can refer to a tool for reviewing and investigatingcommunications. Again, the alerts can represent a potential violation ofa predetermined standard. The conduct surveillance system can generatethe alerts in response to an electronic communication between personsmatching a violation of a predetermined policy. As described in greaterdetail elsewhere in the present disclosure, the predetermined policy caninclude a scenario, a target population, and a workflow.

In some embodiments of the present disclosure, the scenario can includea machine learning classifier. Additionally, in some embodiments of thepresent disclosure, the scenario can include a lexicon. Again, asdescribed herein, the lexicon can represent one or more terms or andregular expressions. A non-limiting example of a term that can beincluded in the lexicon is a string of one or more text characters,(e.g. a word).

At step 144, the system can determine whether determining whether eachof the at least one alert represents an actual violation of thepredetermined policy. As a non-limiting example, if the predeterminedpolicy can configured to detect the dissemination of confidentialinformation. This could represent a violation of a law, regulation, orinternal policy. But a communication identified by the predeterminedpolicy as a potential violation may not represent an actual violation ofthe underlying law, regulation or policy (i.e. a false positive). Insome embodiments of the present disclosure, determining whether eachalert represents an actual violation of the policy is referred to as“actioning” the alert. This can include determining whether each of theat least one alert represents an actual violation of the policy, law, orethical standard that the policy/scenario that generated the alert isconfigured to detect. Actioning the alert can include displaying thealert to a user and receiving a user input from a user interfacerepresenting whether the alert represents an actual violation of thepolicy.

In some embodiments of the present disclosure, the scenario can includea machine learning classifier and determining whether the at least onealert represents an actual violation can include labeling the alert andusing the labeled alert to train the machine learning classifier. Asanother non-limiting example, the present disclosure contemplates thatlabeling can include labeling alerts as “good” “bad” and “neutral.”Optionally, a “good” alert is an alert that is considered to correctlyidentify a violation (e.g. a compliance risk), a “bad” alert is an alertthat does not correctly identify a violation (i.e. a false positive),and a “neutral” alert is an alert that is not a true or false positive.This can include alerts where there is ambiguity, or insufficientinformation to determine whether an alert is correct at the time that itis reviewed.

At step 146, the system calculates a metric based on the actualviolations and the potential violations where the metric can include anumber of false positives in the at least one alert or the number offalse negatives in the at least one alert. In some embodiments of thepresent disclosure, the system can display the metric to the user of thesystem.

At step 148, the system can change the scenario, the target population,and/or the workflow based on the calculated metric. If the scenario usedby the system includes one or more lexicons, changing the scenario caninclude adding or removing one or more terms or regular expressions fromthe lexicon(s). In some embodiments of the present disclosure, thetarget population includes a domain exclusion list and changing thetarget population includes changing the domain exclusion list.

The present disclosure also contemplates that, in some embodiments, theelectronic communication can include metadata, and the scenario caninclude rules for filtering the communication based on the metadata.When the scenario includes rules for filtering the communication basedon the metadata, changing the scenario can include changing the rulesfor filtering the communications based on the metadata.

The following describes actioning, and in particular aspects of how auser can “action” an alert, in accordance with some embodiments of thepresent disclosure. Some examples are illustrated in FIG. 9. In someembodiments, according to actioning at the hit level, a user can actiona single hit under an alert, enabling more accurate reviews and granularfeedback loops (see FIG. 13, for example). For actioning at the alertlevel, all hits under an alert can also be actioned. Also, to action atthe communication level, all hits on a communication can be actioned.Thus, actioning, or an action can refer to a label that is applied to asingle hit, all hits under an alert, and/or all its on a message.

In some embodiments, when marking a message as having been reviewed, thereviewed status is part of the default status list. If a new status listis created, then the reviewed status will not be available unless it ismanually added. In some embodiments, when escalating a hit, alert, ormessage, people of interest can be assigned. In some embodiments,multiple communications can be actioned from the list. Actioning fromthe list view applies resolved status changes to hits containing an openstatus.

Regarding assignments, according to some embodiments, a hit, alert, ormessage can be assigned to another user, which will be displayed andaccessible from their dashboard. Group assignments can also be donewithin escalation workflow. For instance, LDAP (lightweight directoryaccess protocol) groups can be assigned during the escalation workflowfor a hit, alert, and/or message. To change alert status, in someembodiments the hit statuses for a particular alert can be overwritten,once all hits are resolved or unresolved. In some embodiments, if anyhits for a particular alert remain open, the alert actions may onlyapply to that open hit.

Regarding actioning status configurations, in some embodiments,functional permissions are available for actioning, thereby controllinga user's ability to action single messages or multiple messages at once.In some embodiments, a case management API includes actioning at the hitlevel in addition to actions at the alert and message level. Regardingassignment of manual alerts, in some embodiments, a manual alert can beassigned at the message level for an individual message. Manual alertscan be distinguished from system-generated alerts via the person icon inthe alert pill. In some embodiments, to support supervision workflow,alerts may be segregated.

Now particularly referring to the diagram of FIG. 13, in someembodiments, according to actioning at the hit level, alerts andcorresponding actioning can serve as an input into reporting andimproved model training. A system is shown (and its operational flow),according to one embodiment of the present disclosure. The systemprovides for a feedback loop such that the results of reviewed alertscan be fed back to components of the system that are used for furthertraining of models (as well as creating and evaluating lexicons,creating scenarios, and creating policies).

In some embodiments of the present disclosure, the system shown in FIG.13 can be configured to implement one or more of the methods describedwith reference to FIGS. 1A-1C. As shown in FIG. 13, the system caninclude modules for creating and evaluating models and lexicons 1302,modules for creating scenarios 1304 and modules for creating policies1306. In some embodiments of the present disclosure, these three modulescan be used alone or in combination to perform the methods describedwith respect to FIG. 1B. These three modules 1302 1304, and 1306 can becollectively referred to as “cognition studio” or a “scenario builder.”Optionally, the repository 1308 can be used to store scenarios and/orinformation about the alerts, models, or labeled data that are describedwith reference to FIGS. 1A-1C above.

Similarly, as shown in FIG. 13, the system can include modules forgenerating alerts 1310, reviewing alerts 1312, and labeling hits 1314.In some embodiments of the present disclosure, these modules can beconfigured to perform part or all of the methods illustrated anddescribed with reference to FIGS. 1A and 1C. Additionally, FIG. 13 alsoillustrates a feedback path 1316 for how labeled hits can be fed backinto “cognition studio” to further improve the scenarios created.Optionally, the present disclosure contemplates that the feedbackillustrated in FIG. 13 is the feedback described above with reference toFIGS. 1A and 1C.

With reference to FIG. 22, a user interface for accessing a repositoryis shown (e.g. the cognition repository 1308 illustrated in FIG. 13) isshown. The user interface can allow a user to browse, search, import,and export models, lexicons, scenarios, and any of the other data storedin the repository. The exported models, lexicons, scenarios, or otherdata can be referred to as “artifacts.”

With reference to FIGS. 23A-23F, user interfaces for configuring ascenario according to embodiments of the present disclosure are shown.FIG. 23A illustrates a user interface for viewing one or more datasets.FIG. 23B illustrates a user interface for labeling a dataset. FIG. 23Cillustrates an annotation applied to a dataset and an interface forapplying labels to a dataset. FIG. 23D illustrates a user interface forconfiguring a lexicon to be applied to the dataset. FIG. 23E illustratesa user interface for evaluating a lexicon. FIG. 23F illustrates ascenario created using the lexicon that was configured in the interfaceshown in FIG. 23E.

Through the use of a series of functional tools for creating andevaluating lexicons, creating scenarios, and creating policies (labelledcollectively in the diagram as “Cognition Studio”), a user (such as adata scientist) user can create a model (e.g., perform training of amodel) in cognition studio for evaluation against established datasets.The user can then create scenarios based on the model(s), lexicons, andnon-language features (NLF). Next, the user can create polic(ies) whichmap to the scenario(s) and population.

Following the steps collectively labeled under “Cognition Studio”, auser such as a business analyst publishes the scenario(s) to a datarepository labeled in the diagram of FIG. 13 as “Cognition Repository”.The repository can be a data storage device that provides forversion-controlled storage of all models, lexicons, scenarios, andpolicies, and which can allow for labeling of active or draft versions.A user such as a system administrator can select relevant scenario(s)and can select a target population. The user can also select targetcommunication types (e.g., chat, email, etc.) and channels (e.g., chatapplications, email servers, etc.), and mark the policy as active.

The system according to some embodiments can then use a new activepolicy or policy version against all newly ingested electroniccommunications to generate alerts as appropriate (see label in FIG. 13of “Cognition Logic”) and as described in further detail above. Next, inoperations collectively labeled as “Alert” in the diagram, a user suchas an analyst (e.g., compliance representative, etc.) can review thegenerated alerts and label each hit according to, for instance,escalation workflow in which a true positive is identified. The labeledhits can then be used as feedback to the “Cognition Studio” forsupervised improvement of the aspects discussed above with respect tothese components and respective functions.

Example Computing System Architecture

FIG. 12 is a computer architecture diagram showing a general computingsystem capable of implementing one or more embodiments of the presentdisclosure described herein. A computer may be configured to perform oneor more functions associated with embodiments illustrated in, anddescribed with respect to, one or more of FIGS. 1-11 and 13-26. Itshould be appreciated that the computer may be implemented within asingle computing device or a computing system formed with multipleconnected computing devices. For example, the computer may be configuredfor a server computer, desktop computer, laptop computer, or mobilecomputing device such as a smartphone or tablet computer, or thecomputer may be configured to perform various distributed computingtasks, which may distribute processing and/or storage resources amongthe multiple devices.

As shown, the computer includes a processing unit, a system memory, anda system bus that couples the memory to the processing unit. Thecomputer further includes a mass storage device for storing programmodules. The program modules may include modules executable to performone or more functions associated with embodiments illustrated in, anddescribed with respect to, one or more of FIGS. 1-11 and 13-26. The massstorage device further includes a data store.

The mass storage device is connected to the processing unit through amass storage controller (not shown) connected to the bus. The massstorage device and its associated computer storage media providenon-volatile storage for the computer. By way of example, and notlimitation, computer-readable storage media (also referred to herein as“computer-readable storage medium” or “computer-storage media” or“computer-storage medium”) may include volatile and non-volatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer-storageinstructions, data structures, program modules, or other data. Forexample, computer-readable storage media includes, but is not limitedto, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memorytechnology, CD-ROM, digital versatile disks (“DVD”), HD-DVD, BLU-RAY, orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bythe computer. Computer-readable storage media as described herein doesnot include transitory signals.

According to various embodiments, the computer may operate in anetworked environment using connections to other local or remotecomputers through a network via a network interface unit connected tothe bus. The network interface unit may facilitate connection of thecomputing device inputs and outputs to one or more suitable networksand/or connections such as a local area network (LAN), a wide areanetwork (WAN), the Internet, a cellular network, a radio frequencynetwork, a Bluetooth-enabled network, a Wi-Fi enabled network, asatellite-based network, or other wired and/or wireless networks forcommunication with external devices and/or systems.

The computer may also include an input/output controller for receivingand processing input from a number of input devices. Input devices mayinclude, but are not limited to, keyboards, mice, stylus, touchscreens,microphones, audio capturing devices, or image/video capturing devices.An end user may utilize such input devices to interact with a userinterface, for example a graphical user interface on one or more displaydevices (e.g., computer screens), for managing various functionsperformed by the computer, and the input/output controller may beconfigured to manage output to one or more display devices for visuallyrepresenting data.

The bus may enable the processing unit to read code and/or data to/fromthe mass storage device or other computer-storage media. Thecomputer-storage media may represent apparatus in the form of storageelements that are implemented using any suitable technology, includingbut not limited to semiconductors, magnetic materials, optics, or thelike. The program modules may include software instructions that, whenloaded into the processing unit and executed, cause the computer toprovide functions associated with embodiments illustrated in, anddescribed with respect to, one or more of FIGS. 1-11 and 13-26. Theprogram modules may also provide various tools or techniques by whichthe computer may participate within the overall systems or operatingenvironments using the components, flows, and data structures discussedthroughout this description. In general, the program module may, whenloaded into the processing unit and executed, transform the processingunit and the overall computer from a general-purpose computing systeminto a special-purpose computing system.

CONCLUSION

The various example embodiments described above are provided by way ofillustration only and should not be construed to limit the scope of thepresent disclosure. Those skilled in the art will readily recognizevarious modifications and changes that may be made to the presentdisclosure without following the example embodiments and applicationsillustrated and described herein, and without departing from the truespirit and scope of the present disclosure.

What is claimed is:
 1. A computer-implemented method, comprising:receiving at least one alert from a conduct surveillance system, whereinthe at least one alert represents a potential violation of apredetermined standard and wherein the conduct surveillance systemgenerates the alerts in response to an electronic communication betweenpersons matching a violation of a predetermined policy, wherein thepredetermined policy comprises a scenario, a target population, and aworkflow; determining whether each of the at least one alert representsan actual violation of the predetermined policy; calculating a metricbased on the actual violations and the potential violations wherein themetric comprises a number of false positives associated with the atleast one alert or the number of false negatives associated with the atleast one alert; and changing at least one of the scenario, the targetpopulation, or the workflow based on the calculated metric.
 2. Thecomputer implemented method of claim 1, wherein the scenario comprises amachine learning classifier, and wherein determining whether the atleast one alert represents an actual violation comprises labeling the atleast one alert and using the labeled at least one alert to train themachine learning classifier.
 3. The computer implemented method of claim1, wherein the metric is displayed to a user.
 4. The computerimplemented method of claim 1, wherein the scenario comprises a lexicon,and wherein the lexicon represents one or more terms or regularexpressions.
 5. The computer implemented method of claim 1, whereinchanging the scenario comprises changing the lexicon by adding orremoving terms or regular expressions from the lexicon.
 6. The computerimplemented method of claim 1, wherein, in response to determining thatthe at least one alert represents an actual violation, actioning thealert.
 7. The computer implemented method of claim 6, wherein actioningthe alert comprises receiving a user input from the user interfacerepresenting whether the at least one alert represents an actualviolation.
 8. The computer implemented method of claim 1, wherein thetarget population comprises a domain exclusion list and wherein changingthe target population comprises changing the domain exclusion list. 9.The computer implemented method of claim 1, wherein the electroniccommunication comprises metadata, the scenario comprises rules forfiltering the electronic communication based on the metadata, andwherein changing the scenario comprises changing the rules for filteringthe electronic communications based on the metadata.
 10. A system,comprising: at least one processor; at least one memory storing computerreadable instructions configured to cause the at least one processor toperform functions for creating and/or evaluating models, scenarios,lexicons, and/or policies, wherein the functions include: receiving dataassociated with at least one of text data, model training, lexicons,scenarios, and policies, wherein the functions for creating and/orevaluating models comprise creating at least one scenario based on atleast one of the models, lexicons, and non-language features; creatingone or more policies mapping to the at least one scenario and apopulation; upon receiving an alert that a policy match occurs,triggering an alert indicating, to a user, that a policy match hasoccurred which requires a user action, wherein a policy corresponds toactions that violate at least one of a combination of signals andmetrics, a population, and workflow.
 11. The system of claim 10, whereinthe model training comprises training at least one model configured toanalyze the text data from one or more electronic communications betweenat least two persons.
 12. The system of claim 10, wherein the useraction comprises review and interaction by a user via a user interface.13. The system of claim 10, wherein the model training comprisesevaluating the model against established datasets.
 14. The system ofclaim 10, wherein the alert to the user is evaluated by the user and acorresponding user decision is made to confirm or deny accuracy of thealert.
 15. The system of claim 14, wherein the user decision is providedinto a feedback loop, and wherein the feedback loop is configured toimprove the model training.
 16. The system of claim 15, wherein the userdecision is provided into the feedback loop and wherein the feedbackloop is configured to improve the lexicons, scenarios, or policies. 17.The system of claim 16, wherein the feedback loop is configured tochange a lexicon.
 18. The system of claim 17, wherein changing thelexicon comprises configuring the lexicon so that it includes orexcludes terms or regular expressions.
 19. The system of claim 15,wherein the feedback loop is configured to measure the rate of falsepositives and to change one or more of the lexicons, scenarios, andpolicies based on the rate of false positives.
 20. The system of claim15, wherein the scenario includes Boolean operators, and wherein thefeedback loop is configured to change one or more of the Booleanoperators.
 21. The system of claim 16, wherein the feedback loop isconfigured to monitor the rate of false positives over a period of time,and change one or more of the lexicons, scenarios, and policies based onthe rate of false positives over the period of time.
 22. Anon-transitory computer-readable medium storing instructions which, whenexecuted by at least one processor of a computer, perform functions thatinclude: receiving at least one alert from a conduct surveillancesystem, wherein the at least one alert represents a potential violationof a predetermined standard and wherein the conduct surveillance systemgenerates the alerts in response to an electronic communication betweenpersons matching a violation of a predetermined policy, wherein thepredetermined policy comprises a scenario, a target population, and aworkflow; determining whether each of the at least one alert representsan actual violation of the predetermined policy; calculating a metricbased on the actual violations and the potential violations wherein themetric comprises a number of false positives associated with the atleast one alert or the number of false negatives associated with the atleast one alert; and changing at least one of the scenario, the targetpopulation, or the workflow based on the calculated metric.